Conversation
posix_lifetime_lock_probe opened a throwaway fd on the lock file to confirm an already-claimed lock, then closed it. fcntl(2) record locks are scoped to (process, inode), so that close silently released the real lock too, even though the reservation's own fd stayed open. Check the in-process claim before opening anything and return early; that branch only ever needed to trust the registry. Signed-off-by: Amir Fathi <amirfathi.me@gmail.com>
|
Thanks for opening this — it has been seen, and it is queued. This note is automated, but it is not a brush-off: it exists so you know where your PR stands instead of having to guess from silence. Current review status: working through a backlog. What that means for this PR, concretely:
Things that will genuinely speed it up whenever review does happen:
If this fixes a bug, a reproduction we can run is worth more than a description of the symptom. Thanks for contributing, and sorry in advance for the wait. |
DeusData
added
bug
|
Thank you for isolating the claim-lock descriptor path and providing a separate-process regression. This affects coordination between processes, so we need more time for the lifecycle review before making an integration decision. It should not be treated as proof that every cold-start race in #2162 is resolved. |
DeusData
left a comment
DeusData
left a comment
There was a problem hiding this comment.
Thank you, @AmirF194, for this one. It is a genuinely subtle catch, and the separate-process regression makes it easy to verify. Here is the lifecycle review we promised on the 19th.
The diagnosis is right. The lifetime lock is our only fcntl record lock, and POSIX releases those when any descriptor to the file is closed. The holder's own probe therefore opened a second descriptor, closed it, and quietly gave the lock away. Your early return fixes the probe. Windows is not affected: LockFileEx locks belong to a handle, and that path is not compiled there.
Two small asks before we merge:
- The sibling path in
posix_lifetime_lock_try_acquirehas the same pattern. When this process already holds the claim (process_result == 0), it callsprivate_regular_file_at_is_safe(endpoint->dir_fd, lock_name, 1)(src/daemon/ipc.c:1261on currentmain). That helper opens and closes the lock file (ipc.c:848-857), so it drops the held lock exactly the way the probe did. Could you give it the same no-descriptor treatment, for exampleprocess_result == 0 && endpoint_runtime_still_valid(endpoint) ? 0 : -1? Please add a sibling test too: acquire, calltry_acquireagain in the same process (expect 0), and check that the child still cannot take the lock. - Harden the child's verdict in the test. Report "held" only when
open()succeeded andF_SETLKfailed withEACCES/EAGAIN, and use a distinct value for anything else. At the moment any failure in the child, even a wrong path, looks like a pass.
Optional: a one-line comment at posix_record_lock_set stating the rule: never open and close the lifetime lock file in a process that may hold it, because fcntl locks are per process. The lock is fcntl rather than flock on purpose, so that forked children do not inherit it, which is exactly why the rule has to hold.
For the record: this hardens every process that holds the lifetime lock. The production daemon never probes its own lock, though, so this does not resolve the #2162 cold-start reports or the #2057 cold-storm flake. We will keep those open separately. Thanks again for tracking this down. Findings like this are what make the coordination layer trustworthy.
This branch has not been deployed
2 participants
While chasing #2162 I found a real, separate bug in
posix_lifetime_lock_probe(src/daemon/ipc.c): when the in-process registry says this process already holds the lifetime lock, it still opens a throwaway fd on the lock file just to close it again and return 1.fcntl(2)record locks are scoped to (process, inode), not (fd, inode), so that close silently drops the real lock too, even though the reservation's own fd stays open.Two probes back to back are enough to lose it. I wrote a test that acquires the reservation, probes it twice, then forks a genuinely separate process that tries to take the same lock directly. On main it succeeds, the lock is gone. With the fix it fails as expected.
Fix: check the in-process claim before opening anything and return early. That branch only ever needed to trust the registry, not touch the file.
scripts/test.sh --suites daemon_ipc: the new test fails on main (child_result == 1), 50 passed / 1 skipped with the fix.Not a fix for #2162 itself, I ran into this while reading that code, it's a separate bug.